Amplifying signals of misunderstanding improves coordination in dialogue

We report a dialogue task which investigates how the mechanisms of miscommunication contribute toward referential coordination. Participants communicate via a text-based instant messaging tool which is used to identify turns that were edited prior to sending. These turns are transformed by the server into artificial self-corrections, and sent to the participants. The patterns observed in the dialogues show that these interventions have a beneficial effect on referential coordination.</p

The Mediatisation of the Chinese Dama in Chinese English-Language Media:A Cognitive Linguistic Approach

The term ‘Chinese dama’ was originally coined by the Wall Street Journal in 2013 to refer to a group of middle-aged and elderly Chinese women who, somewhat frenetically, purchased gold or other items. This study employs a cognitive-linguistic approach to critical discourse analysis to examine how Chinese damas are linguistically mediatised in the Chinese English-language news media. A specialised corpus of 41 news articles with 26661 words, covering the years between 2013 and 2019, was built for this purpose. Informed by Maslow’s ‘hierarchy of needs’ theory, four most recurrent themes of Chinese dama news discourses were identified and coded. The analysis of these discourses suggests that whilst there is divergence in how newspapers construe Chinese damas’ participation in social activities when they are agentive, there is convergence in terms of schematising the conflicts between Chinese damas and the other parties. This seems to fit with the media’s ideological framework, steering ultimately towards the legitimisation of excluding Chinese female seniors from the public realm.</p

The PAT annotation model for Multimodal Instructions

We present the development, implementation and application of the PAT annotation model, which describes instructional documents that consist of pictures and text (PAT). In document design research (Schriver, 1997) the combination of pictures and text has been noted but not so much investigated in terms of function and content. Although useful starting points have been provided (Bateman, 2014), we are unaware of a standard methodology to describe and evaluate picture-text relations. This leaves document designers without specific guidelines, while readers and users may experience difficulties in effectively processing multimodal content due to mismatches with their expectations and cognitive capacities. DevelopmentAs the possibilities to describe multimodal documents are infinite, we advocate conducting corpus studies and reader/user studies in tandem to determine the relevance of annotation categories and their values. Based on preliminary analyses of a corpus with first-aid instructions (currently 297), we conducted multiple reader and user studies to investigate the effectiveness of particular design features. With the results of these studies we developed and fine-tuned the specification of 51 types of functional and content relations between textual elements and between text and pictures. ImplementationOur corpus annotation is supported by the PAT Workbench, a custom-made online tool that provides a flexible environment to systematically investigate multimodal designs by facilitating storage, annotation, retrieval and evaluation of documents (See, Figures 1, 2 and 3 and https://cosmo.service.rug.nl/patworkbench/login/). The workbench includes ‘smart’ OCR for uploaded documents, user-defined specification of annotation categories, and a tool for creating gold standard annotations based on multiple annotations. ApplicationAs a worked example, we will present the results of a comparative study that involved the application of the PAT annotation model to a subcorpus of 46 first-aid instructions from two editions of Het Oranje Kruis Boekje 2011 and 2016. Het Oranje Kruis is a Dutch organisation that provides learning materials for first-aid certification trainings. A comparison of multimodal instructions (117 pictures and 9416 words in total) for 23 tasks in both editions of Het Oranje Kruis Boekje allows us to conclude that the two editions are similar in terms of the visualised actions, but differ in terms of: text content (preambles, alternative actions, control information); the type of shot used in the pictures (close-up/medium shot versus long shot); and text-picture relations in terms of layout (alignment versus proximity). Future workThe PAT project (http://www.rug.nl/let/pat) will deliver theoretical results in terms of empirically validated models for effective multimodal presentations and authoring guidelines for multimodal documents. Future work will include more comprehensive textual analysis and finer-grained analysis of the pictorial materials, coverage of a greater number and a wider variety of instructions, (semi-)automatic annotation, more empirical evaluation, and (semi-)automatic generation of potentially effective text-picture combinations for multimodal instructions.BibliographyBateman, J. (2014). Using multimodal corpora for empirical research. In The Routledge handbook of multimodal analysis, pp. 238–252. Routledge, London.<br/